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Abstract 

Self-assessment is vital for online learning since it is one of the most essential skills of distance 
learners. In this respect, the purpose of this study was to understand learners’ self-assessment 
quiz taking behaviours in an undergraduate level online course. We tried to figure out whether 
there is a relation between self-assessment quiz taking behaviours and final exam scores or not. 
In addition, we investigated how self-assessment quiz taking behaviour differs with respect to 
learner profile. In line with this purpose, 677 students’ 6092 test events across Project Culture 
course on Sakai CLE LMS were analyzed. For the analysis of the quantitative data, one-way 
ANOVA, Chi-Square test of independence, independent-samples t-test and descriptive statistics 
were utilized. The results revealed that learners who attended self-assessment quizzes regularly 
had higher final exam scores than others who did not attend those quizzes. Also, they were more 
satisfied with the course than others study field. In addition, learners who attended self- 
assessment quizzes regularly had a higher degree of perceived learning. However, number of 
attempts to those quizzes does not have an effect on final exam scores. On the other hand, a 
statistically significant relationship was found between attempt number and gender in favour of 
female learners. 

Abstract in Turkish 

Oz-degerlendirme, uzaktan ogrenen iyin en temel becerilerden biri oldugundan ^evrimirp 
ogrenmede iyin 90k onemlidir. Bu baglamda, bu yakjmada ogrenenlerin kisa oz-degerlendirme 
testlerine katilma davramjlari arajtirilmtjnr. Oz-degerlendirme testlerine katilma durumunun ders 
bajansinda bir farklilasmaya sebep olup olmadigi ve ogrenen profiline gore nastl degijtigi 
incelenmijtir. (^altjma baglammda Sakai CLE LMS iizerinden sunulan Proje Kultiirii dersi 
kapsamtnda 677 ogrencinin 6092 test olayi analiz edilmijtir. Veri analizinde Tek yonlii varyans 
analizi, Chi-Square testi, bagimsiz orneklem t-testi ve betimsel istatistikler kullamlmijnr. 
Arajttrma sonuylartna gore oz-degerlendirme testlerine kattlan ogrencilerin ders bajarilan 
katilmayanlara gore daha yiiksektir. Ayrtca, oz-degerlendirme testlerine diizenli olarak kattlan 
ogrencilerin ders memnuniyeti ve algilanan ogrenme diizeyleri katilmayanlara gore daha yiiksektir. 
Oz-degerlendirme testlerini tekrarlama sayist ile ders bajarisi arastnda anlamk bir ilijki 
gozlemlenmemigtir. Oz-degerlendirme testlerine katilma davraniji, ogrenenin cinsiyeti ve 
boliimiine gore farkkkk gostermemektedir. Ote yandan oz-degerlendirme testlerini tekrar yapma 
yiizdesi kadm ogrenenler lehine anlamli bir farklikk gostermektedir. 

Keywords: Self-Assessment, Self-Assessment quiz, test event analysis, online course, final exam 
scores, course grade 
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Introduction 

Self-assessment is useful in providing learners with better understanding of the subject matter 
(Claxton, 1995). It leads to motivation (Pintrich & Schunk, 2002; Schunk, 2004), enhance 
learning (Boud, 2000), improvement of results in the final exam (Cassady & Gridley, 2005; 
Cukusic, et al., 2014; Ibabe & Jauregizar, 2009; Wilson, Boyd, Chen, & Jamal, 2011), beneficial 
for improvement of learner engagement (Gikandi, et al., 2011). 

The question addressed in this study is whether the self-assessment quizzes would form any 
differentiation in learning that we can detect from log files. We tried to find out what patterns in 
self-assessment quiz taking could be detected in log files. We focused on the information (test 
event data, learner profile), which we could acquire from log files and a mini survey, which is 
about the perceived learning and satisfaction with the course. We parsed the raw of log files as 
Number of Attempts (No Attempt, Attempt Once, and Multiple Attempts), Self-Assessment 
Quiz Taking Behaviour (No attendance, Irregular attendance, Regular attendance with one 
attempt, Regular attendance with multiple attempts) and learner profile (gender and department). 
After that, we asked the research questions below to detect the behaviours in self-assessment 
quiz taking and to see if final exam scores differ significantly with respect to those behaviours: 

• Do final exam scores differ significantly with respect to self-assessment quiz taking 
behaviour of learners? 

• Do learner satisfaction and perceived learning differ significantly with respect to self- 
assessment quiz taking behaviour? 

• Does self-assessment quiz taking behaviour differ significantly with respect to learner 
profile? 

• Do self-assessment test events differ significantly with respect to learner profile? 

The aim of this study is to understand learners’ self-assessment quiz talcing behaviors in the 
Project Culture online course, which was delivered by one of the biggest and oldest foundation 
university in western Turkey in the fall of 2015, as a part of course evaluation activities to 
improve course quality. 

Background and related work 

Literature on the concept of self-assessment is quite diverse. It is not new and discussed in many 
instructional design textbooks (Gagne, et al., 1988; Gale, 1984; Laurillard, 1993; Rowntree, 1991; 
Taylor, 1998). Self-assessment is described by Oscarson, (1978) as an individual’s ability to 
identify and self-evaluate his/her own skills in a particular area of expertise. According to 
Panadero (2011), “self-assessment is the qualitative assessment of the learning process, and of its 
final product, realized on the basis of pre-established criteria”. According to McMillan and Hearn 
(2008) Self-assessment is a cyclic process as represented in Figure 1, and it occurs when students 
judge their own work to improve performance as they identify discrepancies between current and 
desired performance. 
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Self- Self- 

Monitoring Judgment 


Learning Targets 
and Instructional 
Correctives 


Figure 1. Student self-assessment cycle, (McMillan & Hearn, 2008) 

As well as McMillan and Hearn (2008), Rolheiser and Ross (2013) and Zimmerman (2002) define 
self-assessment as learners’ judgment of their own work, based on evidence and explicit criteria, 
for the purpose of improving future performance. 

There are several characteristics of self-assessment (Heidi & Du, 2007). First, it should base on 
transparent criteria, which can be generated by either instmctor or student or co-generated by 
both the instmctor and the students (Dochy & McDowell, 1997; Garcia & Floyd, 1999; 
Frederiksen & Collins, 1989; Hanrahan & Isaacs, 2001; Wiggins, 1998; Stiggins, 2001). Second, it 
should provide feedback that guides students’ efforts and strategies (Adams, 1998; Paris & Paris, 
2001; Horner & Shwery, 2002; Taras, 2005). Third, it should be ongoing and allow learners 
monitoring themselves regularly (Andrade & Boulay, 2003; Goodrich, 1996; Gikandi et al, 2011). 
In our case, as a limitation, criteria for self-assessment were instructor-generated since allowing 
students to define their assessments criteria was not manageable due to the high enrolment rates 
(approximately 600 students in each semester). Feedback provided for each question in the self- 
assessment quizzes as brief information sharing. In addition, self-assessment quizzes placed in 
each module to provide students with ongoing monitoring of themselves. 

According to Boud and Falchikov (1989), self-assessment is a “student activity through which 
he/she judges their own learning”. Bourke (2010) states that if we look self-assessment from the 
viewpoint of students, the aim of it can be categorized under six types as (a) seeking an opinion 
of others (especially teacher) if they learnt, (b) getting marks and grades as an external measure, 
(c) reflecting on their performance, (d) identifying their role in learning and assessment process, 
(e) setting learning goals and (f) evaluating learning content. In our case, self-assessment quizzes 
were limited with automated versions of type 1 and 2 level due to again large number of students. 
Interaction with each of 677 students to provide an instantaneous reflective feedback for their 
self-assessment was not manageable because of the work load it could create. 

Ibabe and Jauregizar (2009) distinguishes self-testing and self-assessment. Self-testing involves 
students’ checking their performance against provided test items (with right and wrong answers) 
(Boud and Brew (1995) cited in Ibabe and Jauregizar (2009)). The self-assessment quizzes in our 
case could be understood as self-testing tasks, but with feedback. These kind of practice tests 
(Wilson et al., 2011) can improve performance (Balter et al, 2013; Gretes & Green, 2000; Snooks, 
2004), and provide students with the opportunity to review course material, demonstrate 
knowledge and identify weaknesses (Kulik et al., 1984), see instructor’s expectations (Snooks, 
2004), increase students' attitudes positively (Deutsch et al., 2012).There are also disadvantages 
with assessment in online environment (Balter et al., 2013; Zhang et al., 2004). One of them is 
the problem of knowing who is answering the questions and also whether this person is receiving 
help or not (Carter et al., 2003). Another difficulty is that almost correct solutions cannot be 
handled as smoothly as on face-to-face environment (Balter et al., 2013). Although many 
researches highlight the effectiveness of self-assessment in educational settings, as presented 
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above, one of them focuses on test analytics of learners’ in online courses as big data (Cukusic 
et al., 2014). The analysis which is shared in this paper was conducted to investigate how self- 
assessment quiz taking behavior occurs in our system. 

Methodology 

Data was gathered from user sessions of online Project Culture course, which run on Sakai CLE 
LMS. Test Event sessions of learners were extracted from overall course data and analyzed. Test 
event session represents a single instance of a student attempting a particular self-assessment test. 
Each test event session contains username, course code, test ID, numbers of attempts, and score. 
6092 test events of 677 students were analyzed. In addition, data of perceived learning and course 
satisfaction gathered from a mini survey which was utilized at the end of the course. 

Course selection 

Project Culture Course was used as the case in this study. It is a foundation course of the 
university. This course is chosen by the students studying on Bachelor programs (Faculty of 
Science and Letters, Faculty of Fine Arts, Faculty of Law, Faculty of Economics and 
Administrative Sciences, Faculty of Communication, Faculty of Architecture and Faculty of 
Engineering) or on Associate Degree programs (Vocational School). It is a compulsory course 
for students of all departments within each faculty and vocational school program. 

The aim of the course was to teach what were considered to be the basic knowledge of project 
design. The main point is to provide the basic information for students to perform practicable 
projects in the direction of their real experiences and support them with producing new projects. 
We preferred this course because it is an independent study field, it is neither science-centric such 
as calculus or physics, nor social-science-centric such as law or communication. 

Structure and production of the course 

At the beginning of the production process of course; subject matter experts, instructional 
designers, developers and administrators determined the content and structure of the course by 
getting together. It was decided for the course to include 7 modules and each module included a 
50-pages reading material, 20-minutes watching material and a self assessment activity of 10 
questions. In this regard, 50-pages reading material, video text and 100 questions were prepared 
by specialists for each module. Each module was worked on by different specialist and necessary 
royalty was paid to specialists. 350 pages reading material, 140 minutes video and 700 questions 
were produced in total for the course. After contents are prepared, they were reviewed by 
instmctional designers and grammatical editing was provided by editors. Then, instructional 
design was performed and production process started. The process of design and production of 
the course took six months in total. 

Project Culture course is composed of 7 modules. Structure of course and modules in LMS was 
provided in Screenshot 1 (Course Structure), Screenshot 2 (Course Module Stmcture) and Screenshot 3 
(Self-Assessment Test Page). 
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Testier 


Proje Kiiltiiru 

Not Defteri |j 

_______ Bu ders sizlere projenin ne demek olduQunu blr projenin nasil yapilabilecegini fon (finansal destek) sagiayan bir kurum adina proje hazidama mantigmi anlamamza yardim edebilmek adina tasarlanmigtir Bu ders 

yardimiyla basmdan sonuna blr projenin planlanmasi uygun bicimde yazimi ve uygulanmasi asamalanna iltskin bilgl edinirken aym zamanda planli dusunme fikirlerini detaylandirabilme arastirma straleji gellgtirme 
riskleri ongomie fikrini kabul ettirebilme becerilen de kazanmaniz saglanacaktir 

Bu ders bireysel olarak ogrenmenlzi gergeklegtirebilmeniz adina 'okuma parcalan' "video icerikleu" ve destekleyici egitsel elkinlikler'T barindiran 7 modUlden olusmaktadir Her bir modulun icerigine asagidaki 
basliklardan ulasabiBrsiniz ModUller ilgili konuya yonelik olarak okuma, izleme ve degerlendirme elklnlikleri igermektedlr Bu etkinlikleri gercekleslirmeden bir modulu tamamlayip digerini aktif hale geliremezslniz 

Ders igerikleri: 

"ic -J Ders Tanitimi 

Derse baslamadan once bu kismi dikkatlice okuyunuz Bu Ogrenmenizi etkili kilmak adina onemli bir adimdir Ders tamtiminda dersin yapisi islenis sekli degerlendirmesi ve kurallan verilmistir Bu kismi 
okuduktan sonra ogrenme Igeriklerinde verilen asagidaki modullere sirasi lie ulasabilirsiniz 
_ 

Proje Planiama ve Ybnetimine Giris Temel Kavramlar 

★ U M- I.:. 

Proje Dongusu Yonetimi 

*U v_ 

Mantiksal Cerceve Yaklasimi 

★ a v_ 

Sonin. Hedef ve Payday Analizi 

★ U_ 

Faaliyel Planlamasi Kaynak Planlamasi ve Bulce 

*U__ 

Gorunurluk. Surdurulebilirlik ve Rapodama 

*U v__ 

Uygun Fon Kaynaklari Bulma 


Screenshot 1. Course Structure 



5 Ogrenme igerikleri > Modiil 1: 


Yazdirma goriiniimti iqindekiler Sayfasi Geri ileri 


Proje Planiama ve Yonetimine Giri§: Temel Kavramlar 

Bu modulde proje planiama ve yonetimi alanmdaki temel kavramlari. projelerin Ozelliklerini ve projelerin asamalanm goreceksiniz 


§imdi sirasi lie a$a$idakl etkinBklerl gergeklegtirerek modulu tamamlayalim 

Okuyalim: 

/• Proje Planiama ve Yonelimine Giris: Temel Kavramlar 

Konuya hazirlik yapabilmek adina yukandaki kitap boiUmunU okuyalim 



Pekigtirelim: 

Ogrendiklerimizi pekistirmek adina asagidaki degerlendinneyi yapalim 

★ ✓' Modul 1-Etkinllk i. Uegerlendinne 

Degerlendirme bu modulde yer alan konulan Iceren 10 sorudan olusmaktadir 


Screenshot 2. Course Module Structure 
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2 Kisa Smav ve Testier <*e 

Proje Kulturu Ozdegerlendirme Modiil 1 

Geribildirimi Goster igindekiler 


Bolum 1 /1 - 


Soru 1/10 

Projenin planlanan faaliyetlerin takvim yerine gbrsel bir §ekilde temsili ne ile saglamr? 

A. Gantt §emasi 

B. Kritik yol agi 

C. Uriln akif diyagrami 

D. Pareto grafigi 
Sepimi Sifirla 


10.0 Puan 


Geri lleri 


Kaydet pikis 


Screenshot 3. Self-Assessment Test Page 

As mentioned earlier there was one self-assessment quiz at the end of each module. It consists of 
ten multiple-choice questions concerning issues raised by the related module. The quiz questions 
cover all of the main points of modules. For each module a question pool, that contains at least 
100 questions, was created. Learner comes across 10 different random questions at each attempt. 
After each attempt, learner is given automated feedback. Feedback includes a short explanation 
about right and wrong answers and refers to related topic in module. Number of attempts is not 
limited. Online multiple-choice tests were used for self-assessment because of its ease of use for 
both developers and learners. One example for questions and feedback was provided in Screenshot 
4 and 5.Translation of the question, is provided in Table 1: 


Table 1: Translation of the sample question and feedbacks into English 

Question: 

What is a visual representation of a project's planned activities against a 
calendar called? 

a. A Gantt chart 

b. A critical path network 

c. A product flow diagram 

d. A Pareto chart 

Feedback for 

correct answer 

Feedback for 

wrong answer 

Congratulations, You answer the question of Project Management Tools 
chapter right. 

Sorry, that is incorrect! Please review Project Management Tools chapter in 
your book again. 
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$ Kisa Sinav ve Testier a» 

Proje Kiilturu Ozdegerlendirme Modul 1 

Geribildirimi Goster igindekiler 


Bolum 1 /1 - 


Soru 1/10 

Projenin planlanan faaliyetlerin takvim yerine gorsel bir pekilde temsili ne ile saglamr? 

✓ • A. Gantt gemasi 

B. Kritik yol agi 

C. UrOn aki§ diyagrami 

D. Pareto grafigi 

Sepimi Sifirla 

Cevap Anahtari: A 

Geribildirim: Tebrikler, "Proje Yonetim Araplari" konu bagligina ait soruyu dogru cevapladmiz. 


10.0 Puan 


Geri ileri 


Kaydet piki§ 


Screenshot 4. Correct answer 


$ Kisa Sinav ve Testier as 


Proje Kulturu Ozdegerlendirme Modul 1 

Geribildirimi Goster ipindekiler 


Bolum 1 /1 - 


Soru 1/10 

Projenin planlanan faaliyetlerin takvim yerine gorsel bir gekilde temsili ne ile saglamr? 

A. Gantt gemasi 

B. Kritik yol agi 

X • C. UrOn aki§ diyagrami 

D. Pareto grafigi 


Sepimi Sifirla 


10.0 Puan 


Geribildirim: Yanlip cevapladmiz! Kitabimzdaki "Proje Yonetim Araplari" konu bapligim tekrar gozden gegiriniz. 


Geri Ileri Kaydet Qiki§ 

Screenshot 5. Wrong answer 

Self-assessment quizzes were designed in order to provide learners with an opportunity to 
ascertain their mastery in particular topics and to assess their understanding of the module. The 
aim of those was to provide learners with the opportunity to have opinion about their learning. 
Attending this self-assessment activity was voluntary. 


European Journal of Open, Distance and e-Learning - Vol. 19 / No. 2 
ISSN 1027-5207 
© 2016 EDEN 


21 













Self-Assessment Quiz Taking Behaviour Analysis in an Online Course 

Yasin Ozarslan, Ozlem Ozan 


Test event data 

Test event data includes username, course code, test id, test score and number of attempts. 

Test score 

This indicates the number of points a student earned on a self-assessment test. 

Number of attempts 

A test which is embedded in the course site triggers various events. The Event Logs for all tests 
are created and maintained automatically in Sakai LMS. Number of attempts, which is the 
number of retaking a self-assessment test, gathered from those records. In this study, Number of 
Attempts was categorized as No Attempt, Attempt Once, and Multiple Attempts. No attempt case 
occurs when a learner opens the test page but does not take the test. Attempt once case occurs 
when a learner takes the test once. Multiple attempts case occurs when a learner takes the test more 
than once. 

Learner data 

Learner data includes profile data and self-assessment quiz taking behaviour of learners. 

Profile data of learners 

This is the data about characteristics of learners who enrolled the online courses. Profile data 
covers final exam scores, departments, grades and genders of learners. Descriptive statistics of 
learner profile data was represented in Table 2. 

Table 2: Number of students and achievement of students in project culture course 


Gender 

N (677) 

% 

Total Course Grade Average 

Male 

353 

52.1 

72.93 

Female 

324 

47.9 

73.24 



100 

73.08 


Table 3: Enrolment by faculty and vocational school program 


Faculty 

N 

% 

Faculty of Science and Letters 

65 

9.6 

Faculty of Law 

130 

19.2 

Faculty of Economics and Administrative Sciences 

134 

19.8 

Faculty of Communication 

58 

8.6 

Faculty of Architecture 

79 

11.7 

Faculty of Engineering 

122 

18.0 

Faculty of Art and Design 

47 

6.9 

Vocational School 

42 

6.2 

Total 

677 

100 


Self-Assessment Quiz Taking Behaviour 

Although taking self-assessment tests was voluntary and there was no restriction on number of 
attempts, some of the learners did not take any of seven self-assessment tests. Some of them 
took self-assessment tests of some modules. Some of them took the self-assessment tests of each 
module with just one attempt, and some of them took the self-assessment tests of each module 
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with multiple attempts. Therefore, self-assessment qui^ taking behaviour was categorized as four types 
as follows: 

• No attendance: This case means never taking a self-assessment test. It refers to the 
situation that participants did not take any of the self-assessments tests. 

• Irregular attendance: This case occurs when a learner takes self-assessment test of some 
modules, not seven of them. It refers to the situation that participants took some of the 
self-assessments tests. 

• Regular attendance with one attempt: This case refers to the situation that participants 
took all of the self-assessments tests with just one attempt in each test. 

• Regular attendance with multiple attempts: This case refers to the situation that 
participants took all of the self-assessments tests with multiple attempts in each test. 

Descriptive statistics of self-assessment quiz talcing behaviour are represented in Table 4. 


Table 4: Descriptive statistics of self-assessment quiz taking behaviour 


Code 

Self-assessment Quiz Taking Behaviour 

# of Students 

Type 0 

No attendance 

64 

Type 1 

Irregular attendance 

268 

Type 2 

Regular attendance with one attempt 

153 

Type 3 

Regular attendance with multiple attempts 

192 

Total 


677 


Survey data 

A course evaluation survey, which was composed of two Likert scale questions and one open- 
ended question as presented in Table 5, was conducted at end of the course. The aim of course 
evaluation survey was to gather data about satisfaction and perceived learning of learners. 


Table 5: Course evaluation survey 


No 

Category 

Question 

Strongly 

Disagree 

(1) 

Disagree 

(2) 

Neither Agree 
nor Disagree 
(3) 

Agree 

(4) 

Strongly Agree 
(5) 

Q1 

Perceived 

Overall, 1 found this course 
helpful to learn the topic 

□ 

□ 

□ 

□ 

□ 

Q2 

Satisfaction 

1 am satisfied with the 

course 

□ 

□ 

□ 

□ 

□ 

Q3 

Comments 

Do you have any 
suggestions or comments 
to improve course 
content? 

□ 

□ 

□ 

□ 

□ 


Data analysis 

For the analysis of the quantitative data, one-way ANOVA, independent samples t-test, Chi- 
Square test of independence and descriptive statistics were applied. For the normal distribution 
of data, the skewness and kurtosis coefficients were examined. First, for the normality test 
skewness coefficient of a distribution taken in the range of -1.5 to +1.5 and the kurtosis 
coefficient of a distribution taken in the range of -1.5 to +1.5 according to Tabachnick and Fidell 
(2013). Then, a one-way between subjects ANOVA was conducted to compare the effect of self- 
assessment quiz taking behavior on achievement. After that, an independent-samples t-test was 
conducted to compare self-assessment test scores and test attempt. For data of learner profile, 
Chi-Square test of independence was performed. Besides ANOVA, Chi-Square test of 
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independence and independent samples t-test, descriptive statistics such as percentages (%), 
mean scores (X) and frequencies (f) were used as well. 

Limitations 

Participants of this study were college students. They perhaps processed information differently 
than those without college experience would. In addition, there were no foreign students, who 
enrolled to the course, since the course was in Turkish. Therefore, this study should also be 
repeated with different participant groups and cultures. 

Findings 

In this section, the findings obtained in the study are presented under three headings based on 
research questions. 

Do final exam scores differ significantly with respect to self-assessment quiz taking 
behaviour of learners? 

A one-way between subjects ANOVA was conducted to compare the effect of self-assessment 
quiz taking behaviour on achievement in conditions No attendance, Irregular attendance, Regular 
attendance with one attempt. Regular attendance with multiple attempts. Table 6 presents the 
one-way ANOVA results, which demonstrate the significant effect of self-assessment quiz taking 
behaviour on achievement at the p < .05 level for the four conditions [F(3,673) = 4.59, 
p = 0.003]. 


Table 6: One-Way Analysis of Variance of achievement 


Source 

df 

SS 

MS F 

P 

Between Groups 

3 

1765.99 

588.66 4.59 

.003 

Within Groups 

673 

86,302.57 

128.24 


Total 

676 

88,068.56 




* p < .05 


Post Hoc comparisons using the Tukey HSD indicated that the mean score for the No attendance 
condition (M = 69.50, SD = 11.80) was significantly different from the Regular attendance with one 
attempt condition (M = 73.90, SD = 11.98), and the Regular attendance with multiple attempts condition 
(M = 74.88, SD = 11.41). However, the Irregular attendance condition did not significantly differ 
from other conditions. Taken together, these results suggest that learners who attended to self- 
assessment quizzes of each module at least once and more had higher scores on final exam than 
others. 

Do learner satisfaction and perceived learning differ significantly with respect to 
self-assessment quiz taking behaviour? 

87% of learners answered the course evaluation survey as reported in Table 7. 


Table 7: Number of learners who answered the course evaluation survey 



Mean 

# of response 


Perceived learning (Over 5) 

3.76 

591 

87% 

Course satisfaction level (Over 5) 

3.77 

591 

87% 

Number of Suggestions/Comments 


123 

18% 
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A one-way between subjects ANOVA was conducted to compare the effect of self-assessment 
quiz taking behaviour on perceived learning in conditions Irregular attendance, Regular 
attendance with one attempt, Regular attendance with multiple attempts. Table 8 presents the 
one-way ANOVA results, which demonstrate the significant effect of self-assessment quiz taking 
behaviour on perceived learning at the p<.05 level for the four conditions [F(2, 588) = 3.58, 

p = 0.028]. 


Table 8: One-Way Analysis of Variance of Perceived Learning 


Source 

df 

SS 

MS F 

P 

Between Groups 

2 

6.20 

3.10 3.58 

.028 

Within Groups 

588 

508.62 

.86 


Total 

590 

514.82 




* p < .05 


Post Hoc comparisons using the Tamhane’s T2 indicated that the mean score for the Regular 
attendance with one attempt condition (M = 3.64, SD = 0.90) was significantly different from the 
Regular attendance with multiple attempts condition (M = 3.90, SD = 0.85). The Irregular attendance 
condition (M = 3.74, SD = 0.99) did not significantly differ from other conditions. These results 
suggest that learners who attended to self-assessment quizzes of each module at least once and 
more perceived a high level of learning than others. 

A one-way between subjects ANOVA was conducted to compare the effect of self-assessment 
quiz taking behavior on satisfaction in conditions Irregular attendance, Regular attendance with 
one attempt, Regular attendance with multiple attempts. Table 9 presents the one-way ANOVA 
results, which demonstrate the significant effect of self-assessment quiz taking behaviour on 
satisfaction at the p < .05 level for the four conditions [F(2, 588) = 4,40, p = 0.013]. 


Table 9: One-Way Analysis of Variance of Satisfaction 


Source 

df 

SS 

MS F 

P 

Between Groups 

2 

7.45 

3.72 4.40 

.013 

Within Groups 

588 

499.14 

.84 


Total 

590 

506.59 




* p < .05 


Post Hoc comparisons using the Tamhane’s T2 indicated that the mean score for the Regular 
attendance with one attempt condition (M = 3.64, SD = 0.89) was significantly different from the 
Regular attendance with multiple attempts condition (M = 3.92, SD = 0.83). The Irregular attendance 
condition (M = 3.73, SD = 1.01) did not significantly differ from other conditions. These results 
suggest that learners who attend to self-assessment quizzes of each module at least once and 
more were more satisfied than others. 

In course evaluation survey, 18% of learners had some suggestions to improve the course or 
made some comments about the course. 123 entries in total were made. 11 learners evaluated 
self-assessment quizzes in their entries. 64% of them advised to increase the number questions in 
self-assessment quizzes. Learners’ direct quotations were shared in Table 10. 
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Table 10: 

Learner (L) 

LI 

L2 

L3 

L4 

L5, L6 
L7, L8, L9 
L10, Lll 


Learners' direct quotations, which are related to self-assessment quizzes. 

Direct Quotation 

Self-assessment quizzes improved permanent learning for me 
Self-assessment quizzes were helpful 

Changing the questions in each attempt resulted in better learning 

Number of self-assessment quizzes should be increased 

Number of questions should be increased in self-assessment quizzes 

Number of True/False questions should be increased in self-assessment quizzes 

Increasing number of questions, which aim to promote higher order thinking skills, 

could increase student engagement 


Does self-assessment quiz taking behaviour differ significantly with respect to 
learner profile? 

A Chi-square test of independence was calculated to compare the self-assessment quf taking 
behaviour and gender. No significant relationship was found between these variables (X 2 = 4.84, 
N = 677, p = 0.184). 

A Chi-square test of independence was calculated to compare the self-assessment quf taking 
behaviour and their departments. There wasn’t a significant relationship between self-assessment quf 
taking behaviour and their department, (X 2 = 20.43, N = 677, p = 0.494). 

Do self-assessment test events differ significantly with respect to learner profile? 

6092 test events, which belong to 677 learners, across seven modules on Sakai CLE LMS, were 
analyzed as represented in Table 11. 


Table 11: Descriptive statistics of Test Events 



Female 

# of test Events 

Male Total 

% 

Test Score Average 
Female Male Total 

No Attempt 

554 

690 

1244 

20.42% 




Attempt Once 

1640 

1713 

3353 

55.04% 

69.79 

68.33 

69.04 

Multiple attempts 

867 

628 

1495 

24.54% 

71.77 

67.24 

69.87 

Grand Total 

3061 

3031 

6092 

100.00% 





In 20.42% of test events, no attempts were made to take the test as reported in Table 12. In 
57.37% of test events, resulted in just one attempt, learners took the test once. In 22.21% of test 
events, resulted in multiple attempts, learners took the test multiple times. 
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Table 12: Descriptive statistics of Test Attempt Numbers 


Test Attempt 

# of Test Event 

% 

No Attempt 

1244 

20.42 

1 

3495 

57.37 

2 

689 

11.31 

3 

225 

3.69 

4 

142 

2.33 

5 

92 

1.51 

6 

61 

1.00 

7 

39 

0.64 

8 

32 

0.53 

9 

24 

0.39 

10 and above 

49 

0.80 

Total 

6092 

100.00 


An independent-samples t-test was conducted to compare self-assessment test scores in Attempt 
Once and Multiple Attempts conditions. There was not a significant difference in the scores for 
Attempt Once (M = 69.05, SD = 30.47) and Multiple Attempts (M = 69.87, SD = 29.40) 
conditions; t (2964.68) = 0.89, p = 0.371. 

A Chi-square test of independence was performed to examine the relation between the attempt 
numbers and gender. A significant relationship between attempt number and gender was found 
(X 2 = 34.15, N = 4848, p < .001). 58% of multiple attempts were made by female learners. 

Discussion and conclusion 

It is important to understand how self-assessment quizzes affect course achievement in an online 
course. In this regard, this paper analyzed self-assessment quiz taking behavior of learners in 
Project Culture online course, which was delivered by one of the biggest and oldest foundation 
university in western Turkey in fall of 2015. Nikou and Economides (2016) reported that 
computer and mobile based self-assessment increased science learning motivation and 
contributed to better learning achievement. Similarly, Belski, (2010) reported that self-assessment 
could result in better engagement and improve students’ course performance as well as Wilson 
et al. (2011). Similarly, our results suggested that learners who attended to self-assessment quizzes 
of each module at least once and more, had higher scores on final exam than others. However, 
our analysis is not strong enough to claim a correlation or cause-and-effect relationship. We can 
say that there is a relation between taking self-assessment quizzes and final exam scores but we 
cannot state a direction or a cause-and-effect. There could be two possible situations for the 
relation we observed. First, learners who had high self-regulation skills took those quizzes. They 
might be academically more successful students, since they were skillful at self-regulating their 
learning. So, they had higher scores. Second, learners took higher scores since self-assessment 
quizzes helped self-regulation of themselves. It was impossible to track overall learning process 
of the learners with the available log file data. Therefore, we need further research in this topic. 

We observed that learners who attended to self-assessment quizzes of each module at least once 
and more perceived a higher level of learning than others. In addition, they were more satisfied 
than others. These findings are parallel to other findings in the field of self-assessment study; 
according to researchers (Brown & Harris, 2013; McMillan & Hearn, 2008; Oscarson, 1989; 
Panadero & Alonso-Tapia, 2013) correctly implemented self-assessment is by itself a process that 
promotes learning. 
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In this study 57.37% of test events resulted in just one attempt, and no statistically significant 
difference was found in the test scores for Attempt Once and Multiple Attempts. This means 
that although attending self-assessment quizzes regularly implies a difference on final exam 
scores, number of attempts to those quizzes does not have an effect on the scores. Therefore, 
allowing two or three attempts for each self-assessment quiz will probably be sufficient to achieve 
the objectives of the self-assessment. It seems like creating an extra system load by allowing 
unlimited attempts is not necessary. 

No significant relationship was found between self-assessment quiz taking behavior and learner 
profile in terms of their gender and department. However, a significant relationship between 
attempt number and gender was observed. 58% of multiple attempts were made by female learners. 
In her speech on TED Talks, Saujani (2016), the founder of Girls Who Code, gives some 
statistics and states that females are more perfectionist than males since we are raising our girls to 
be perfect while we are raising our boys to be brave. Her detection and intuition are true for 
Turkish culture, as well as American culture. In this respect, female learners might have a 
tendency to reach to the highest score, in other words to reach the perfect score. So, the reason 
of higher percentage rate of multiple attempts in favor of female learners could be this tendency. 
However, this issue requires further research. 

These findings could be references for those who plan to design and develop self- assessment 
activities in online courses. In addition, the results of our analysis could be helpful to policy 
decision makers in order to support their educational development. To improve external validity, 
it is planned to replicate the analyses in next semesters. 
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